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Sir: 

•The final rejection of Claims 1-11 is hereby being appealed, 
which are reproduced in the attached Appendix. 

1. Real Party in Interest 

The real party in interest is U.S. Philips Corporation, the 
assignee herein. 

2. Related Appeals and Interferences 

The Appellant is not aware of any appeals or interferences 
that relate to the present application. 
3. Status of all Claims 

Claims 1-11 were submitted in the original application. Claims 
1-11 were finally rejected in the Office Action dated October 25, 

2000 and are /currently being appealed. 

03/19/2001 HVUQK61 00000106 141870 030(7910 
01 FC:120 310.00 CH J 
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4 . Status of Amendments 

An After Final Amendment mailed January 17, 2001, was filed. 
In an Advisory Action dated February 21, 2001, it was stated that 
this After Final Amendment would be entered upon filing of a Notice 
of Appeal and an Appeal Brief. 

5 . Summary of the Invention 

The present invention is directed to a method and device for 
displaying an image of a scene from a freely selectable viewpoint. 
As can be seen from Figure 1, the invention includes obtaining 
input images of the scene, each from a respective input viewpoint 
14a, b, as described on page 4. As can be seen from Figure 3, the 
invention includes computing model information representing points 
10a, b 11, a, b located in a space of the scene from which input 
information in the input images originate, as described on page 4. 

As can be seen from Figure 4, the invention includes selecting 
the selectable viewpoint 40, as described on page 6. As further 
described on page 6, the invention includes determining for an 
output pixel in the output image which output point represented in 
the model information is visible in the output image at that output 
pixel 42. Further, determining a pixel value of that output pixel 
from the image information in the input image or input images which 
originated from the output point, as described on page 6. Also, 
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displaying 48 the output pixel in the output image according to the 
pixel value, as described on page 8. 

As can be seen from Figure 3, the invention also includes 
computing a respective model for each input image, the respective 
model comprising information about surface patches 10a, b 11a, b 
located in the space of the scene, substantially each surface patch 
corresponding to a respective set of pixels in the respective 
image, as described on page 4. As can be seen from Figure 4, the 
invention includes determining 42 for each respective model, which 
if any, respective point from the surface patches of that 
respective model is visible in the output image at the output pixel 
according to that respective model, as described on page 6. 

The invention also includes selecting 44 the output point from 
the respective points on the basis of comparison of parameters of 
the surface patches containing the respective points in different 
models for which the respective points have been found, as 
described on page 8. As further described on page 8, determining 
the pixel value from the image information in the input image 
corresponding to the respective model from which the output point 
is selected. 

6. Issues Presented for Review 

The first issue is whether Claims 1, 2 and 8 under 35 USC 
102(a) as anticipated by or, in the alternative, under 35 USC 
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103(a) are unpatentable over Chen (Chen et al. 1993, entitled "View 
Interpolation for Image Synthesis", ACM, pages 279-286) . The 
second issue is whether Claims 4-7 and 9-11 under 35 USC 103(a) are 
unpatentable over Chen. The third issue is whether Claim 3 under 
35 USC 103(a) is unpatentable over Chen in view of Debevec (Debevec 
et al., entitled "Modeling and Rendering Architecture from 
Photographs' 7 , Computer Graphics Proceedings, Annual Conference, 
1996) . 

7 . Grouping of the Claims 

The Appellant respectfully submits that claims 1-11 either 
stand or fall together. 

8 . Arguments 

Claims 1, 2 and 8 stand finally rejected under 35 USC 102(a) 
as anticipated by or, in the alternative, under 35 USC 103(a) as 
being unpatentable over Chen. Claims 4-7 and 9-11 stand finally 
rejected under 35 USC 103(a) as being unpatentable over Chen. 
Claim 3 stands finally rejected under 35 USC 103(a) as being 
unpatentable over Chen in view of Debevec. 

In order to make a proper obvious rejection under 35 U.S.C. 
103, MPEP Section 706.02 (j) requires that the prior art reference 
(or references when combined) must teach or suggest all of the 
claim limitations. Further, either the references must expressly 
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or impliedly suggest the claimed invention. Ex parte Clap, 227 
USPQ 972, 973 (Bd. Pat. App. & Inter. 1985) 

In view of the above, it is respectfully submitted that Chen 
neither expressly nor impliedly suggest al of the features recited 
in the claims. In particular, such features include "computing a 
respective model for each input image, the respective model 
comprising information about surface patches located in the space 
of the scene, substantially each surface patch corresponding to a 
respective set of pixels in the respective input image 1 ', as recited 
in Claims 1 and 8 . 

In addressing the above feature in the above rejections, page 
2, left column, lines 14-21 of Chen is being relied on. In this 
portion, Chen discloses pixel correspondence can be established by 
if range data and the camera transformation are available. 

However, in page 2, left column, lines 5-12, Chen discloses: 

"Our method uses the camera's position and orientation and the 
range data of the images to determine a pixel-by-pixel 
correspondence between images automatically. The pairwise 
correspondence between two successive images can be pre-computed 
and stored as a" pair of morph maps. Using these maps, 
corresponding pixels are interpolated interactively under the 
user's control to create in-between images." 

In view of the above disclosure, it is evident that that the 

pixel correspondence disclosed on page 2, left column, lines 14-21 

of Chen is between two successive images. Therefore, it is evident 
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that Chen does not disclose "computing a respective model for each 
input image", as required by the claims. 

In the Office Action dated October 25, 2000, the Examiner 
stated that Chen does disclose creating a model of the scene in 
terms of pixels with range data. In response, the Applicant 
respectfully submits that even if the Examiner's interpretation of 
Chen is correct, it still does not read on the above mentioned 
feature . 

Claims 1 and 8 require "obtaining input images of a scene, each 
from a respective viewpoint" and then "computing a respective model 
for each input image". Therefore, the claims require computing a 
respective model for each of the different input images of scene. 

However, in page 2, left column, lines 2-7, Chen only discloses 
to determine a pixel-by-pixel correspondence between images 
automatically." Based on this disclosure, it is evident that Chen 
does not disclose "computing a respective model for each input 
image", as required by the claims. Therefore, it is respectfully 
submitted that this feature is distinguishable over Chen. 

It is also respectfully submitted that Chen neither expressly 
nor impliedly suggests "selecting the output point from the 
respective points on the basis of comparison of parameters of the 
surface patches containing the respective points in different 
models" as recited in claims 1 and 8. In addressing this feature 
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in the above rejection, page 3, left column, lines 46-61 of Chen is 
being relied on. 

However, in page 3, left column, lines 46-61, Chen only 
discloses : 

"To generate an in between view of pair of images, the offset 
vectors are interpolated linearly and the pixels in the source 
image are moved by the interpolated vector to their destination. ..The 
interpolation is an approximation of the transformation of the 
pixel coordinates by a perspective viewing matrix..." 

Based on the above disclosure, it is evident that Chen does 
not disclose "selecting the output point from the respective points 
on the basis of comparison of parameters of the surface patches 
containing the respective points in different models", as required 
by the claims. Therefore, it is respectfully submitted that this 
feature is also distinguishable over Chen. 

The above-described deficiencies of Chen are also not 
addressed by Debevec since this reference is being relied on for 
other features. Thus, it is respectfully submitted that the 
invention of claims 1-11 is not obvious over Chen alone or in 
combination with Debevec. Accordingly, the Appellant respectfully 
requests that the final rejection of these claims be reconsidered 
and reversed. 
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Please charge the fee of $300.00 to Deposit Account 
No. 14-1270. 



Respectfully submitted, 
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Attorney 
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• # 

APPENDIX 

1. A method of displaying an output image of a scene from a 

freely selectable viewpoint, the method comprising the steps of 
t - obtaining input images of the scene, each from a respective 
input viewpoint; 

y - computing model information representing points located in a 
space of the scene from which input information in the input' images 
originates ; 

* - selecting the selectable viewpoint; 

X - determining for an output pixel in the output image which 

output point represented in the model information is visible in the 

output image at that output pixel; 
m - determining a pixel value of that output pixel from the image 

information in the input image or input images which originated • 

from the output point; 

* - displaying the output pixel in the output image according to 
the pixel value, 

computing a respective model for each input image, the 
respective model comprising information about surface patches 
located in the space of the scene, substantially each surface patch 
corresponding to a respective set of ^pixels in the respective input 
image; 
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determining for each respective model which, if any, 
respective point from the surface patches of that respective model 
is visible in the output image at the output pixel according to 
that respective model, 

selecting the output point from the respective points on the 
basis of comparison of parameters of the surface patches containing 
the respective points in different models for which the respective 
points have been found; 

determining the pixel value from the image information in the 
input image corresponding to the respective model from which the 
output point is selected. 

2. A method according to Claim 1, wherein for selecting the 
output point from the respective points, preference is given to 
respective points with smaller distance to the selected viewpoint. 

3. A method according to Claim 1, wherein for selecting the 
output point from the respective points, preference is given to 
respective points with smaller difference between a first and 
second angle between a normal of the surface patch to which the 
respective point belongs and lines of sight to the respective 
point, from the selected viewpoint and the respective viewpoint of 
the' input image corresponding to the respective point respectively. 
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4- A method according to Claim 1, for selecting the output 

point, preference is given to respective points, with smaller 
difference between a first and second size of a projection of the 
output pixel in the output image and an input pixel in the input 
image corresponding to the respective point respectively, when the 
output pixel and the input pixel are back-projected onto the 
surface patch. 

5. .A method according to Claim 1, wherein for each 

particular input image an ordered list of intervals in the output 
image is computed, the intervals corresponding to segments of 
respective surface patches successively visible along a scan-line 
in the output image, the scan-line containing the output pixel, the 
output point being determined by determining the interval in which 
the output pixel lies, and wherein the ordered list is computed by 
processing the line-segments successively in an order in which they 
are visible along a path in the particular input image, a 
representation being kept of a continuous range along the scan-line 
spanned by the intervals of processed line-segments, a 
foreground/background position of a line-segment being determined 
by comparing the line-segment to an auxiliary line segment 
extending between points in the model corresponding to the 
endpoints of the continuous range. 
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6. A method according to Claim 1, wherein altered pixel 
values are computed for pixels of the input images, so as to 
represent the effect of lighting changes, the method comprising 

selecting a source point in the scene and a lighting 
distribution image located relative to the source point, a 
respective lighting pixel value being associated with each light 
pixel in the lighting distribution image; 

determining for each respective model which, if any, 
respective lighting point from the surface patches of that 
respective model would be visible in the lighting distribution 
image at a light pixel according to that respective model, 

selecting an alterable point from the respective lighting 
points on the basis of comparison of parameters of the surface 
patches containing the respective lighting points in different 
models for which the respective lighting points have been found; 

altering the image information in the input image 
corresponding to the respective model from which the alterable 
point is selected at an alterable pixel corresponding to the 
alterable point. 

7. A method according to Claim 1 comprising 

obtaining further input images from a further scene, each from 
a respective further viewpoint in combination with a respective 
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further models comprising information about surface patches located 
in a space of the further scene, substantially each surface patch 
corresponding to a respective set of pixels in the further input 
images- 
designating a collection of further surface patches from the 
further models; 

selecting a position and attitude for the collection of 
further surface patches relative to the further scenes- 
determining respective auxiliary viewpoints relative to the 
further scene so that each of the respective auxiliary viewpoints 
has a same spatial relation to the further surface patches as a 
respective one of the viewpoints when the further surface patches 
are located according to the selected position and attitude; 

computing a set of auxiliary images of the further surfaces in 
the collection from the auxiliary viewpoints and computing 
auxiliary models describing the further surface patches in the 
collection as far as they correspond to sets of pixels in the 
auxiliary images; 

replacing a pixel value and model information in the input 
images by a pixel value and auxiliary model information from the 
auxiliary images prior to computing the output image when a depth 
from the respective viewpoint of the input image to a point in the 
scene represented by the pixel value is greater than a depth from 
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the corresponding auxiliary viewpoint to a visible point on a 
further surface patch from the collection. 

8 - A device for displaying an output image of a scene from a 

freely selectable viewpoint, the device comprising 

memory for storing input images of the scene, each from a 
respective input viewpoint; 

memory for storing model information representing points in 
the scene from which input information in the input images 
originates ; 

means for selecting the selectable viewpoint; 

means for determining for an output pixel in the output image 
which output point represented in the model information is visible 
in the output image at that output pixel; 

means for determining a pixel value of that output pixel from 
the image information in the input image or images which originated 
from the output point; 

a display unit receiving the pixel value for displaying the 
output pixel in the output image according to the pixel value, 
wherein the memory for storing model information is arranged to 
store a respective model for each input image, the respective model 
comprising information about surface patches located in the space 
of the scene, substantially each surface patch corresponding to a 
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respective set of pixels in the respective input image; the device 
comprising 

means for determining for each respective model which, if any, 
.respective point from the surface patches of that respective model 
is visible in the output image at the output pixel according to 
that respective model, 

means for selecting the output point from the respective 
points on the basis of comparison of parameters of the surface 
patches containing the respective points in different models for 
which the respective point has been found; 

means for determining the pixel value from the image 
information in the input image corresponding to the respective 
model from which the output point is selected. 
9- A device according to Claim 8 wherein altered pixel 

values are computed for pixels of the input images, so as to 
represent the effect of lighting changes, the device comprising 

means for selecting a source point in the scene and a lighting 
distribution image located relative to the source point, a 
respective lighting pixel value being associated with each light 
pixel in the lighting distribution image; 

means for determining for each respective model which, if any, 
respective lighting point from the surface patches of that 
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respective model would be visible in the lighting distribution 
image at a light pixel according to that respective model, 

means for selecting an alterable point from the respective 
lighting points on the basis of comparison of parameters of the 
surface patches containing the respective lighting points in 
different models for which the respective lighting points have been 
found; 

means for altering the image information according to a 
lighting model in the input image corresponding to the respective 
model from which the alterable point is selected. 
10. A device according to Claim 8, wherein the means for 

determining for each model which output point is visible determine 
for each particular input image an ordered list of intervals in the 
output image is computed, the intervals corresponding to segments 
of respective surface patches successively visible along a 
scan-line in the output image, the scan-line containing the output 
pixel, the output point being determined by determining the 
interval in which the output pixel lies, and wherein the ordered 
list is computed by processing the line-segments successively in an 
order in which they are visible along a path in the particular 
input image, a representation being kept of a continuous range 
along the scan-line spanned by the intervals of processed 
line-segments, the foreground/background position of a line-segment 
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being determined by comparing the line-segment to an auxiliary line 
segment extending between points in the model corresponding to the 
endpoints of the continuous range. 

11. A device according to Claim 8, comprising means for 

obtaining further input images from a further scene, each from 
a respective further viewpoint; 

designating a collection of further surface patches from the 
further models; 

selecting a position and attitude for the collection of 
further, surface patches relative to the further scene; 

determining respective auxiliary viewpoints relative to the 
further scene so that each of the respective auxiliary viewpoints 
has a same spatial relation to the further surface patches as a 
respective one of the viewpoints when the further surface patches 
are located according to the selected position and attitude; 

computing a set of auxiliary images of the further surfaces in 
the collection from the auxiliary viewpoints and computing 
auxiliary model information representing depth of the further 
surface patches in the collection from the auxiliary viewpoints; 

replacing a pixel values and model information in the input 
images by a pixel value and auxiliary model information from the 
auxiliary images prior to computing the output image when a depth 
from the respective viewpoint of the input image to a point in the 
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scene represented by the pixel value is greater than a depth from 
the corresponding auxiliary viewpoint to a visible point on a 
further surface patch from the collection. 
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